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Abstract. Given the probabilities of N messages and a number D of output symbols, a prefix free 
code of minimal redundancy assigns to each message a code string of variable length over alphabet 
[1..-D] such that no code is prefix of another, and the average length of a code is minimized. We 
describe an algorithm computing an optimal prefix free code from unsorted positive integer weights 
in time linear in the input encoding size. Our solution takes advantage of non algebraic instructions, 
such as computing the logarithm integer part and indirect addressing, both supported in constant time 
in modern architectures. This result improves on the complexities of known solutions in the algebraic 
model, 0(N \ogN) through Huffman's original proposal or 0(kN) through Belal et al.'s proposal 
(where k is the minimal number of distinct code lengths of some optimal prefix free code for this set of 
weights) ; and over the reduction to sorting in the RAM model which yields an 0(N log log N) algorithm. 
Potential applications include compression (JPEG, MP3, MPEG) and transmission solutions. 



Keywords: Optimal Prefix Free Code, Minimal Redundancy, Huffman Code, Counting Sort, Linear Time 
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1 Introduction 



Given N positive integer weights W[l..iV] coding for the probabilities { r -i }ie[i ..n] 01 N message^, 

2—, j=i \3\ 

and a number D of output symbols, an Optimal Prefix Free Code is a set of N code strings of variable lengths 
L[1..N] and on alphabet [1..D] such that no string is prefix of another (i.e. Yli=i D~ L ^ = 1) and the average 
length of a code is minimized (i.e. X^iLi £[*]W[i] is minimal). The advantage of such variable length codes 
is that even though the code strings assigned to the messages can differ in lengths (assigning shorter ones to 
more frequent messages yields compression), the prefix free property insures a non-ambiguous decoding. 

Such codes, known since 1952 [T5], are used in "all the mainstream compression formats" from 
mp3 music files to JPEG image files |5I31) and many forms of electronic communications. The concept of 
Optimal Prefix Free Code "is one of the fundamental ideas that people in computer science and data 
communications are using all the time" (Knuth [24]); it "continues to enjoy widespread use in data 
compression programs" and "enjoys a prominence enjoyed by only a relatively small number of 
fundamental methods" (Moffat et al. [IE]); it is "one of the enduring techniques of data compression. 
It was used in the venerable PACK compression program, authored by Szymanski in 1978, and remains no 
less popular today" (Moffat et al. [20]V 

The original description of the code by Huffman [13] yields a heap-based algorithm performing 0{ N log N) 
algebraic operations. Van Leeuwen described a reduction in O(N) algebraic operations [16j to sorting the 
weights, which yields a O(iVloglogiV) solution using non algebraic operations [TT]. Belal et al. [3, described 
an algorithm claimed to perform O(kN) algebraic operations for the unsorted binary case (i.e., when D = 2), 
where k is the number of distinct code lengths for some optimal binary prefix free code. 
Those results yield the following question: 

Can we compute an optimal prefix free code in time less than 0(min{kN, AT log log JV}), for instance 
using non algebraic operations? 

We answer in the affirmative: 

There is an algorithm which computes an optimal prefix free code in time linear in the input encoding 
size, based on non algebraic operations. 

We describe our techniques on the simplest version of the problem, the computation of optimal binary prefix 
free codes (D = 2), in Section [2] 

1. An optimal prefix free code assigns code lengths within a term of 1 to weights within a factor of less 
than 2: W[j] < W[i] < 2W[j] => L[i] < L[j] + 1. (Lemma [J) 

2. Our variant of Counting Sort |15) groups the weights into sorted clusters of constant logarithmic integer 
parts in base 2, in time linear in the encoding size of the input. (Lemma [2]) 

3. Our implementation of the select operator partitions the weights of a cluster around a pivot specified 
by its rank, and updates the partial sums, all in time linear in the size of the cluster. (Lemma [3]) 

4. Our TopDown algorithm computes the variations of code lengths in an optimal prefix free code in time 
0(N). (Lemma H toil) 

5. The variations of the code lengths yield the code lengths themselves via a simple partial sum algorithm 
running in linear time, and the code lengths in turn yield the code strings in time linear in the number 
N of weights. (Theorem Q} 

We describe then how our techniques apply to other problems in Section [3] such as D-ary optimal pre- 
fix free codes for D > 2 (Theorem[2]), a compressed data structure for optimal prefix free codes (Corollary!!]) 
and an algorithm adapted to external memory (Corollary [2]). We discuss in Section |4]thc relevance of our 
results, let it be in practice (Section l4.1|) or in comparison with past (Section I4.2[) and future (Section I4.3j) 
results. 

We note = {i, i + 1, . . . , j} the integer range from i to j, and ,A[i..j] = LA[i], A[i + 1], . . . , A[j]} the set of 

values of an array A which indices occupy this range. 
2 Huffman |T3] introduced the terminology of messages as input and symbols as output, which should not be confused 
with the terminology, used by some other publications, of input symbols, letters or words for the input and output 
symbols or bits (in the binary case). 
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2 Binary Prefix Free Codes 



2.1 Weight Ratio vs Code Lengths Variations 

Moffat et al. [2T] noted that messages of equal weights can be grouped to speed-up the process. Consider 
a given prefix free code and three messages x, y and z such that x and y have the exact same weight 
W[x] = W[y] but are assigned code lengths L[x] < L[y], and such that the codes assigned to y and z differ 
only in their last digit (intuitively, they are "siblings" in the code tree corresponding to the code) . Then, the 
code can be improved by increasing by one the code length assigned to x, and decreasing the code lengths 
assigned to y and z, the smallest of the two being reduced by one and the longest being reduced to 




Fig. 1. A (suboptimal) code-tree where two weights (16 
and 18) within a factor of 2, are assigned code lengths (3 
and 1 respectively) differing by more than 1. 




Fig. 2. Figure [TJs code-tree improved (made optimal) by 
moving up the heaviest child of the parent of the leaf of 
weight 16, to make it a sibling to the leaf of weight 18. 



The same reasoning goes if z corresponds to a set of messages instead of a single one (intuitively a subtree 
in the code tree corresponding to the code) , and if the weights of x and y are merely within a factor of less 
than 2, rather than being equal: Figures [1] and [2] show a simple example. This property directly implies a 
bound on the difference of code lengths in an optimal code between such pairs of messages: 

Lemma 1 (Weight Ratio vs Height). Consider an optimal binary code assigning code lengths L[1..N] 
to messages of positive integer weights W[1..N]. Then W[i] < W[j] < 2W[i] implies that L[i] < L[j]+1, 

Vi,je[l..iV]. 

Proof. Let T be the binary tree corresponding to an optimal code; x\ any node of weight w\ and at l\ 
edges from the root; X2 any node of weight wi < 2wi and at I2 edges from the root; d = l\ — h the 
difference of depths between x\ and X2] 23 the sibling of x\ (hence at h edges from the root), of weight W3; 
max = max{u;i, W3} and x max the node of weight max; and min = min{wi, W3} and x m ± n the node of weight 
min. Define T 1 the tree obtained from T such that 

— the subtree of T rooted in X2 is replaced by a node of weight W2 + max and of children X2 and a; max ; 

— the subtree of T rooted in the parent of w\ and W3 is replaced by a node of weight min and of children 

^min, 

If T is a binary tree corresponding to a prefix free code, then so is T". The variation of redundancy from T 
to T' is (J2 + l)(w2 + max) + (^ — l)min = u> 2 + max(l — d) — min, given that d = l\ — Z 2 . The binary prefix 
free code is optimal if and only if this variation is always non negative, independently of the choice of the 
nodes, which implies that d < t " 2 ~ mln + 1. Since w 2 < 2w 1 < 2max, ^ < 2 and so d < 2 - sis. Given that 

' 1 — max ^ ^ — ' max max 

min > and max > (as all weights do), we get d < 2 and the desired result. 
2.2 Sorting by Logarithm Integer Parts 

Within a group of weights within a factor of less than 2 of each other, Lemma Q] implies that the set of code 
lengths assigned to the corresponding messages is at most of size two. An easy way to build such groups of 
weights is to partition the weights by the integer part of their logarithm in base 2, such that all weights 
within a given cluster are within a factor of less than 2 of each other. Figure [3] shows an example of weights, 
their clustering by integer part of their logarithm, and the same weights sorted by their value. 
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Unsorted: 


{2, 1, 18, 2, 1,2, 16, 1,9, 8, 2, ■■■ ,2,1,-" , 1} 


[log 2 J -Sorted: 


19 10 

({l,-- ,1},{2,-- - ,2},{9,8},{18,16}) 


Sorted: 


19 10 

(!,-•• ,1,2,.-. ,2,8,9,16, 18) 



Note: () denotes an ordered sequence, {} an unordered set, 
and v, ■ ■ ■ , v a sequence or set of x values v. 

Fig. 3. The various states of a list of weights. 

For each weight x, compute [log 2 a;J , the integer part of its logarithm in base 2: this takes N op- 
erations. Since the weights are integers, their logarithm is at least zero. Since all values are between 
and Llog 2 maXj g n jyn W[«]J, the algorithm Counting Sort [15] sorts the TV weights in time and space 
0(N + [log 2 maXjgH jv] W[*]J)j which can be non linear in TV if maXj e ri.,jv] W[i] is larger than 2 N . 

We describe a variant of Counting Sort which runs in time linear in the input encoding size, in particular 
when some weights are larger than the maximal value that one machine word of space can hold, so that the 
input is encoded in more than N machine words. 

Lemma 2 (LogSort). Given N integer weights W[1..N], there is an algorithm which sorts those weights 
by the integer part of their logarithm in base 2 in time linear in the input encoding size. 

Proof. Let w be the number of bits in a machine word. Note that w does not occur in the complexity of our 
algorithm: the algorithm is linear in the input encoding size even if the weights do not hold in O(l) machine 
words. 



Algorithm 1 LogSort(M / [], 1, N) 

1. Compute in LIP[1..JV] the N integer part of the logarithm in base 2 of each of the weight. 

2. Compute the minimum m = minjgn.jv] LIP[i] and maximum M = maXj e ii..Ar] LIP[£] values in LIP. 

3. Initialize to a bit vector B of M — m + 1 bits. 

4. Set B[l] = 1 if 3i € [1..JV] such that LIP[i] = I. 

5. Index B to support the rank and select operators in constant time [10] on the M ~^" +1 word borders. 

6. Compute the number S < N of bits set to 1 in B. 

7. Initialize to an array C of S integers. 

8. Count the number of occurrences of each rounded logarithmic value I, using the rank operator to map positions 
in B to positions in C. 

9. Compute the partial sums of C. 

10. Reposition the values of W according to those partial sums. 



The algorithm first computes the integer parts of the logarithm in base 2 of the weights in the array LIP 
(which name stands for "Logarithm Integer Parts"). It computes the range of those logarithmic values via 
the variables m and M, respectively the smallest and largest integer parts, and maps the values occuring 
in LIP into the range [m..M] via the bit vector B of length M — m + 1. Through this mapping, it counts 
the numbeJl 8 of values occuring at least once in the array LIP and ignore all other values. The rest of the 
algorithm is standard Counting Sort: the partial sums of the counters indicate, for each value marked in B, 
the position of the first occurrence of this value in the sorted array: a simple scan repositions all the values 
in their sorted order. 

To analyze the complexity of this algorithm, note that Step 3 performs M ~" +1 £ 0(log(max 
operations while all other steps have complexity within 0(N). 

1. Computing LIP[l..iV] takes N operations. 

3 5 being the Greek letter for d, stands here for "diversity" in integer part logarithmic value. 
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2. Computing m and M takes |iV comparisons. 

3. Initializing to the bit vector B of size M — m + 1 bits, takes M ~™ +1 operations using word RAM 
parallelism. 

4. Scanning LIP[l..iV] to set B[l] = 1 if 3i e such that LIP[i] = Z, takes A operations. 

5. Indexing i? to support the rank and select operators in constant time [TU] on the M ~" +1 word bor- 
ders uses 0(N log 2 log 2 AT/ log N) C o(A r ) extra bits of space and takes time within 0( M ~™ +1 ) C 
C(NbWeights). 

6. Computing the number 5 < N takes constant time using the rank operator previously indexed. 

7. Initializing to an array C of S integers takes S operations. 

8. Counting the number of occurrences of each rounded logarithmic value /, using the rank operator to 
map positions in B to positions in C, takes in A operations. 

9. Computing the partial sums of C takes (5<A operations. 

10. Repositioning the values of W according to those partial sums, takes A operations. 

This yields a final complexity within 0{N + (log max |^j4j)/u>). 

This complexity is linear in the size of the input: In the (usual) cases where each weight of the input 
holds into a machine word, M — maxjgn j\n Ll°g2 G 0(w), so that the complexity of the algorithm is 

within 0(N), where the number of machine words used to encode the input is exactly N. In the (extreme) 
cases where some weights do not hold in a machine word, we express the complexity of our algorithm in 
function of the number of machine words used to express the input, which is then larger than N. 

Assuming that each weight is encoded separately, the input uses at least ~}2 \\og 2 (W[i])] bits, or J] P^O^M)! > 
N machine words. Both are larger than the algorithm's complexity, 0(N + 8/w) = 0(N+ (log max w\j\)/ w )- 
Hence the algorithm's complexity is linear in the size of the input in both cases. 

Figures |4] to [8] show the values taken by the variables used by the algorithm in the order they are 
computed, as a graphical representation of the execution of the algorithm on the same instance than in 
Figures [U [2] and El 
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Fig. 4. Weights and Logarithm Integer Parts. 
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Fig. 5. Bit vector marking which values appear. 
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Fig. 6. Counters and their partial sum. 
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Fig. 7. Weights [log 2 J -sorted into clusters. 
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Fig. 8. Positions of the first weight of each cluster. 
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Note that although Kirkpatrick et al. Q3] also used the logarithm of the weights (actually, of the normal- 
ized probabilities) in order to batch them in their parallel algorithm, they considered only the sorted case, 
and it is not clear how this results relates to theirs. 

2.3 Operator Select 

Sorting the weights by their logarithm integer part is not sufficient to compute an optimal prefix free code 
through Van Leeuwen's algorithm |16j . Our algorithm, presented in the next section, partitions the weights 
inside each cluster around one single pivot, described by its rank among the elements of the cluster. We 
describe here how to perform this selection and partition the interval in linear time. 

In 1961 Hoare |T2| defined the operator select(W,l,r, p) for l,r £ an d p G which returns 

the element from W[Z..r] such that p — I elements are strictly smaller than it in this range, and r — I + 1 — p 
are larger than or equal to it. A particular case of the select operator is the computation of the median, 
defined by median(W, I, r) = select(W / , I, r, r ~ l 2 +1 ) for l,r£ [1..A]. Blum et al. [4 designed a deterministic 
algorithm to support this operator in a number of comparisons linear in the worst-case. Schonhage et al. [25] 
improved the computational complexity of the median to 3iV + o(N) , and later Dor and Zwick 7 improved 
it further to 2.9427V + o(N). 

The inverse of the select operator is the rank(W, I, r, x) operator for (,r£ and x G [0..J7], which 

returns the number of elements strictly smaller than x, plus one (effectively the position of x in W once 
sorted, if x is in W). The main interest of the select operator is to later partition the input into two parts, 
such that all elements of the first part are smaller than x and all the elements of the second part are larger 
than or equal to x. For this purpose we use a particular implementation: 

Lemma 3 (select operator). Consider N weights W[1..N] given in some arbitrary order, and the vector 
S of the N partial sums S[i] = Y^]=i G on those weights in this order, and an integer interval 

[l..r] C [1..-/V]. Given an integer rank p G [l~r], there is an algorithm select(W,l,r,p) which selects the 
(p)-th's smallest weight x (counting with repetitions) within W[l..r] in G(r — I) algebraic operations. The 
algorithm updates the partial sums while partitioning the weights from W[l..r] into W[l..p— 1] if strictly 
smaller than x and into W[p + l..r] if larger than or equal to x, all of it in G(r — I) algebraic operations. 

Proof. There is a large choice of algorithms to compute the pivot |4l25l7j in time linear in the size of the 
range. Partitioning according to this pivot and updating the partial sums also takes linear time, as no 
elements nor partial sum outside of this interval is affected by the partitioning. 

2.4 TopDown Recursive Algorithm 

Shanon-Fano codes [9127] are such that when the messages are ordered by their weights, the binary codes 
are in lexicographical order, with codes of same code length corresponding to consecutive weights. This 
property, called numerical sequence, yields some techniques to encode the code in less space and access it in 
less time [2616] . Huffman's code does not have this property, but in 1964 Schwartz et al. [25] showed how to 
compute from a Huffman's code, in 0{N) additional time, a canonical code with the same redundancy as 
Huffman's code and the numerical sequence property. 

We give here an algorithm which maintains at all time the numerical sequence property [6], and as such 
directly builds a canonical code [26]. We describe the process recursively for simplicity, even though it is 
intrinsically iterative. Given the partition of the N weights into 5 clusters, the TopDown recursive function 
described in Algorithm [5] computes for each cluster the splitting point where the weights change code lengths 
and reorders the weights around it. The code length is later deduced by summing right to left the variation 
values. 

The essential difference between our algorithm and the one from Shanon and Fano [9 27 (which is also 
"top-down" ) is that each recursive call TopDown( NbWeights, NbClusters, NbRoots) does not necessarily yield 
a prefix free code, but rather NbRoots prefix free codes, which cover the NbWeights messages concerned by 
this particular recursive code. This is essential to the optimality of the code produced by our algorithm, as 
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opposed to the Shanon-Fano algorithm where each recursive call can return only one prefix free code. On 
sorted weights its time complexity would not improve on existing solutions, but in the general case when the 
weights are unsorted it keeps all weights associated with the same code length consecutive in the array, so 
that their sum can be easily maintained. 



Algorithm 2 TopDown(NbWeights, NbClusters, NbRoots) 



if NbClusters = 1 then 
MinLen^Uog, 2™J; 

Increment Variation[NbWeights] by MinLen; 
SplitPoint = 2 * (NbWeights-(NbRoots*2 MinLen )); 
select(W / [], 1, NbWeights, SplitPoint); 
Increment Variation[SplitPoint]; 
else 

CIStart = ClThreshold[NbClusters]; 
CISize = NbWeights - CIStart + 1; 
MinLeaf = miiijg[ Clstart ..NbWeights] W[i]; 
SumWeights = PartialSum[ClStart - 1]; 
MaxNbNodes = [SumWeights/MinLeaf ] ; 

MinLen=Llog 2 ( cls ^+— deS )Ji 

NbPairs = MaxNbNodes + CISize - NbRoots * 2 MlnLen ; 
Variation[NbWeights] = MinLen; 
if 2 * NbPairs < MaxNbNodes then 
/ CIStart -1, \ 
TopDown NbClusters — 1, ; 

V MaxNbNodes— NbPairs / 

else 

SplitPoint = ClStart+ 2*NbPairs- MaxNbNodes; 
select (WfJ, CIStart, NbWeights, SplitPoint); 
Increment Variation[SplitPoint]; 

/ CIStart -1, \ 
TopDown NbClusters — 1, ; 

V MaxNbNodes J 

end if 
end if 



Algorithm [5] receives as parameters 

— NbWeights £ [l.-^V], the number of weights to be considered by the recursive call, corresponding to the 
range of values W[l.. NbWeights]; 

— NbClusters £ [l..<5], the number of clusters to be considered by the recursive call, corresponding to the 
range of values ClThreshold[l.. NbClusters]; and 

— NbRoots £ [1..-/V], the number of prefix free codes required to cover the NbWeights concerned by the 
recursive call (i.e. the number of trees required in the forest corresponding to the output code(s)). 

The algorithm accesses the variable CIThreshold, a vector indicating the first position of each log-cluster, 
without modifying it, and it modifies the following global variables through border effect: 

— Wfl.JV], the weights partially sorted by increasing integer logarithm value; 

— PartialSum[l..iV], the partial sums of W[l..iV] such that PartialSum[i] = Y?j=i and 

— Variation[l..iV], a vector initially set to zero, such that Vari at ion[i] = L[i] — L[i + 1], Vi £ where 
L[N + 1] = (note that L[] is decreasing, so that Variation^] > 0). 
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An overview of the internal variables used by the algorithm aids in understanding the process of the 
algorithm: 

— MinLen is the minimum code length assigned within the current cluster, the other code length being 
MinLen + 1; 

— SplitPoint is the position of the first weight of code length MinLen, the previous weights being of longer 
code length; 

— CISize counts the weights in the current cluster; 

— MinLeaf is the minimal weight within a cluster; 

— SumWeights is the sum of the weights which will be assigned recursively potentially larger code lengths; 

— MaxNbNodes is the maximal number of nodes of weight larger than MinLeaf that the weights considered 
recursively could form; and 

— NbPairs is the number of pairs of weights that need to be formed in order to round the number of nodes 
at level MinLen to the closest power of two inferior. 



Each recursive call is processing exactly one cluster, and dividing it in at most two groups of weights, 
corresponding to one code length each, both within a term of 1 in accordance with Lemma [1] When the 
number 5 of clusters is small, the most costly operation performed by the algorithm is the reordering of the 
weights around a single pivot in each cluster. On the other hand, the number 5 of clusters is never more than 
the number TV of weights, so that even for large values of 5, when the reordering of the weights inside each 
cluster takes only linear time, the overall complexity is linear in the total number of weights N. 

Lemma 4 (Complexity of the Algorithm). Given N weights grouped in 5 clusters of weights within 
a factor of less than 2 of each other, the recursive function TopDown described in Algorithm [H runs in 
O(N) algebraic operations, 5 computation of logarithmic values in base 2, and 0{N) overall time. It uses 
constant space in addition to the space used by its input (constituted of the weights W[1..N], the partial 
sums PartialSum[l..A] and the description of the clusters ClThreshold[l..NbClusters] ) and output (the 
description Variation[l..A] of the variations of the code lengths). 



Proof. The only operations performed by the algorithm which take more than constant time are the select 
operations in each branch of the outermost conditional test and the recursive call to TopDown in each branch 
of the innermost condition test. 

Each select operation is performed on a single cluster: 

— in the first branch of the outermost condition, there is only one cluster, of range [L.NbWeights]; 

— in the second branch of the outermost condition, the range [ClStart..NbWeights] corresponds to the clus- 
ter NbClusters starting at position ClThreshold[NbClusters] = CIStart and finishing at the position 
ClThresb.old[NbClusters + 1] = NbWeights. 

Assume that ClThreshold[<5 + 1] = N for simplicity. As the recursive process merely iterates through the 
clusters, the total complexity is within 0^^ lusters ciThreshold[i + 1] — ClThreshold[z]), which sums to 
O(N). 

Note that each value of the vector PartialSum could easily be computed online in the algorithm, saving a 
linear space in the algorithm at the cost of N additional additions and a sligthly more complicated algorithm 
(see Section HTTjl . 
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For ease of reading (dividing the proof into chewable bits) , we separate the proof of correctness of the 
code produced (Lemma [5]) from the proof of its optimality (Lemma [6]) : 

Lemma 5 (Correctness of the Code). The recursive function TopDown described in Algorithm^ updates 
the vectors of the weights W[1..N], and of the vector of code length variations Variation[l..iV] so that 
J2iLi 2 ~ L[i] = NbRoots, where L[i] = J2f=i Variation[i] . 



Proof. The only instructions affecting the code lengths output are those dealing with the vector Variation 
encoding the variations of the code lengths, each modification concerning only values within the last cluster. 
Increasing any value in the portion Variation[ClStart..NbWeights] of this vector is increasing the code 
lengths assigned to the weights in the other clusters, if any. We prove the correctness of the code produced 
by induction on the number NbClusters of clusters. 

Consider first the base case where NbClusters = 1. The algorithm assigns no more than two distinct 
code lengths, MinLen and MinLen + 1. Each of the NbRoots roots yields at least 2 MlnLen codes, summing 
to NbRoots * 2 MlnLen over all roots. Since NbWeights — (NbRoots * 2 MlnLen ) nodes are paired by increasing 
their code length by one by the instruction "Increment Variation[SplitPoint];", in total SplitPoint = 
2 * (NbWeights — (NbRoots * 2 MlnLan )) nodes are assigned a code of length MinLen + 1 and NbWeights — 
SplitPoint = NbRoots * 2 MlnLen — NbWeights nodes are assigned a code of length MinLen, which yields 
£ 2 - L M = £SpiitPoint-i 2 _(„ inLe n+i) + J^g^^ 2~ Ki ^ = 2 x (NbWeights - (NbRoots * 2 MinLen ) x 
2-MinLen+i + 3 x Nb Roots2 MlnLen ) - 2 x NbWeights2 MinLen = NbRoots, by the definition of SplitPoint. 

Suppose now that NbClusters > 1 and that the code returned by the function is correct for any recursive 
call on less than NbClusters clusters. 

In the case where (MaxNbNodes + ClSize)/NbRoots is a power of 2, the induction hypothesis implies that 
the recursive call to TopDown returns a cover of the NbWeights weights by NbRoots optimal prefix free codes 
of code lengths L'[l.. CIStart — 1] such that Y^i*** - 2 _L 'M = MaxNbNodes — NbPairs. The increment in 
code length is equal to MinLen = log 2 ( clsiz ;+ R M o a o x t K s bHodes ) so that 

N ClStart-1 

>— (L'[t]+MlnL«n) 



i=l 

JV 

V y 2— MinLen 
i=ClStart 



CIStart — 1 

= 2- MinLen x ( ^2 2 ~ L ' li] + CISize) 

i=l 

= 2~ MinLen x (MaxNbNodes - NbPairs + CISize) 

= 2- MinLen x (NbRoots x 2 MinLen - CISize + CISize) 

= NbRoots, 

which implies the correctness of the code in this case. 

In the case where (MaxNbNodes + ClSize)/NbRoots is not a power of 2, the induction hypothesis implies 
that the recursive call to TopDown returns a cover of the NbWeights weights by NbRoots optimal prefix 
free codes of code lengths L'[l.. CIStart - 1] such that Yh^i^ 2~ l 'W = MaxNbNodes. The code length 
increment on [SplitPoint.. NbWeights] is equal to MinLen = log 2 ( C1Slze ^R a x t M s M ° des ) , and to MinLen + 1 on 
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the range [l..SplitPoint — 1], so that 



N 



CIStart — 1 



-LI 



2 _ (- L '[ i ]+ MinLen + 1 ) 

:Poin 

E 



SplitPoint— 1 



^(MinLen+l) 



i=ClStart 
NbWeights 



E 



^SplitPoint 



2-(MinLen+l) 



= 2 



(MaxNbNodes 
+NbWeights - CIStart + 1 
+NbWeights - SplitPoint + 1 

/ 2MaxNbNodes - 2NbPairs 
l v +2NbWeights - 2ClStart + 2 

■KmLan ( MaxNbNodes — NbPairs 

\ +NbWeights - CIStart + 1 

2~ MinLen (MaxNbNodes - NbPairs + CISize) 

2 -MinLen( NbRoots x 2 MinLen) 

NbRoots, 



hence the correctness of the code in this second case. 



We show next that the algorithm TopDown produces NbRoots optimal prefix free codes which roots hold 
weights within a factor of two of each other. In the particular case where NbRoots = 1, this corresponds to 
an optimal prefix free code in the regular sense. 

Lemma 6 (Optimality of the Code). The recursive function TopDown described in Algorithm^ reorders 
the weights W[1..N] and updates the code length variations Variation[l..iV] so that y^,-_ 1 L[i]W^i] is mini- 
mized, where L[i] = Variation[z]. 



Proof. We prove the optimality of the code produced by the algorithm by induction over the number 
NbClusters of clusters in the partition. 

Consider first the base case where NbClusters = 1. Since all the weights are within a factor of two, 
there can be only two distinct code lengths assigned. Given the number of weights and roots required, those 
code lengths can only be the largest integer smaller than and the smallest integer larger than logi ^fjf£~- 
The smallest weights should receive the largest code lengths: the algorithm defines through Variation a 
heap-shaped tree with the SplitPoint longest branches on the left, and aligns via the select calls the 
SplitPoint smallest values as leaves of these branches, which yields the minimality of ^™ ei s hts 

For the case where NbClusters > 1, suppose as an induction hypothesis that the recursive call to 
TopDown (one in each conditional branch of the algorithm) returns a cover of the NbWeights weights by 
NbRoots optimal prefix free codes such that ^^ S 1 tart L[i]W[i] is minimal. The algorithm first reduces the 
NbWeights weights from the input to MaxNbNodes + CISize nodes within a factor of two of each other, 
by computing the number CISize of weights which are already within a factor of two of the minimum 
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weight MinLeaf (since they are in the same cluster). It then computes the sum SumWeights of the remaining 
NbWeights — CISize weights; the number MaxNbNodes of nodes of weights within a factor of two of MinLeaf 
which will be required to cover the weights of the range [L.ClStart — 1]; and the minimum code length 
MinLen to be assigned to one of the MaxNbNodes + CISize nodes (the other nodes, if any, being assigned a 
code length of MinLen + 1) in order to cover them by NbRoots codes. 

If the number MaxNbNodes + CISize of nodes is a power of two, the optimality of the code returned is 
reduced to the optimality of the code returned on CIStart — 1 weights and NbClusters — 1 clusters, assigning 
a constant code length of MinLen to those MaxNbNodes + CISize nodes of weights all within a factor of two. 
More generally, if MaxNbNodes+ClSize is not a power of two, NbPairsMaxNbNodes+ClSize-NbRoots*2 MinLen 
of them are paired reducing this set to NbRoots * 2 MlnLen nodes: the process is similar to the one in the first 
branch of the outermost conditional test, partitioning the nodes into two code lengths within a term of one. If 
the first code where the code length changes is within the leaves (second branch of the innermost conditional 
test), a select operation is performed to insure that the heaviest weights within the cluster are assigned the 
shortest code lengths. Otherwise (first branch of the innermost conditional test), the constraints is simply 
transmitted to the recursive call through the number NbRoots = MaxNbNodes — NbPairs required. In both 
cases the optimality of the code output by the algorithm is reduced to the optimality of the code output by 
the recursive call, which implies the minimality of ^™ ei s hts 



2.5 Final Algorithm 

The results of Sections 12.21 to I2T41 combine to give the following result: 

Theorem 1. Given N unordered positive integer weights W[1..N], there is an algorithm which computes an 
optimal binary prefix free code for those weights in time linear in the input encoding size. 

Proof. Given N unsorted weights, our algorithm goes as follows: 

1. Sort the weights by the integer part of their logarithm in base 2, into clusters of weights all with the 
same logarithmic integer part (Lemma [2]) in time linear in the input encoding size; 

2. Compute a vector of variations in code lengths via the recursive function TopDown(NbWeights, NbClusters, 
NbRoots) with the parameter values of NbWeights = N, NbClusters = S and NbRoots = 1, executed in 
time linear in the number N of weights; 

3. Compute the code lengths assigned to the weights by summing the variations from the last position to 



4. Compute the codes themselves from the code lengths, using the technique described by Schwartz et 
al. |26) in time linear in the number N of weights. 

Thus, the overall complexity of our algorithm is linear in the input encoding size. 
3 Further Results 

The technique described in the previous section yields several other results of interest, which we describe 
more quickly. 
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3.1 JD-ary Prefix Free Codes 



Even though Huffman 13; originally defined the optimal prefix free codes for an arbitrarily large output 
alphabet of size D, most of the subsequent work has focused on the binary case |16I3I26I18I20| . 

Yet Moura et al. [22] showed that optimal D-ary prefix free codes have their own interest to compress, 
to index and to search texts in natural language. They considered codes produced with values of D = 128 
or D — 256 to compress and index texts in natural language where each message is a word of the text. 
They showed experimentally that the use of such codes yields a minor loss in compression: this should be 
expected as a D-ary code is more granular than an optimal binary prefix free code. Most interestingly, they 
showed that such codes also yield much faster results in terms of speed of search in the compressed text: as 
the symbols composing each code are using half a byte (for D = 128) or a byte (for D — 256) in memory, a 
memory alignment which provides a considerable speed-up in practice. 

We show here that those codes can be computed in time linear in the input encoding size using techniques 
only slightly more sophisticated than the ones described for binary prefix free codes. 



Weights ratio vs Code Lengths: Note first that if D messages whose weights are within a factor of D 
are necessarily assigned code lengths within a term of 1 by an optimal D-ary prefix free code, two messages 
whose weights are within a factor of D are not necessarily assigned code lengths within a term of 1 by an 
optimal D-ary prefix free code. 

But even though we consider a larger number of symbols, it is sufficient to cluster the weights still by 
the integer part of their logarithm in base 2 (as opposed to base D), because two messages whose weights 
are within a factor of 2 are assigned code lengths within a term of 1 by an optimal D-ary prefix free code: 

Lemma 7. Consider an optimal D-ary code assigning code lengths L[1..N] to messages of positive integer 
weights W[1..N]. Then W[i] < W[j] < 2W[i] implies that L[i] < L[j]+l, Vi, j e [1..N]. 

Proof. The proof is very similar to the proof of Lemma [1] Let T be the binary tree corresponding to an 
optimal code; x\ any node of weight w\ and at l\ edges from the root; Xi any node of weight u>2 < 1w\ and 
at I2 edges from the root; d = l\ — I2 the difference of depths between x\ and X2- Instead of pushing up the 
minimum out of the two children of the parent of x\ , push up the D — 1 children of maximal weight of the 
parent, hence reducing their code length by a term of d — 1. Given that W2 < 2wi, simply pushing up X\ 
or a node of larger weight is already improving the code when d > 1, pushing up d — 2 other nodes is only 
further improving the code. 

D-ary TopDown: The main difference between our binary and D-ary solutions is in the implementa- 
tion of the recursive function DAryTopDown, taking an additional parameter NbSymbols and called through 
DAryTopDown( NbWeights, NbClusters, NbRoots, NbSymbols), described in Algorithm^ Note that the al- 
gorithm features now two types of logarithms: a logarithms base 2 for the clustering in linear time and a 
logarithms in base D for evaluating the codelenths required for clusters. 

The complexity of this algorithm is linear, by the same argument as the binary case, and the proof of 
the correctness and optimality of the code produced is also very similar. 

Lemma 8. The recursive function DAryTopDown described in Algorithm^ reorders the weights W[1..N], and 
updates the code length variations Variation[l..iV] in time linear in the input encoding size so that 

1. J2iLi L[i]W[i] is minimized and 

2. J2?=i D ~ m = NbRoots, 

where L[i] — ^2f =i Variation^]. 
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Algorithm 3 DAryTopDown(NbWeights, NbClusters, NbRoots, NbSymbols) 



if NbClusters = 1 then 

»jr • r , I i NbWeights | 

MmLen <- Llo gnbSymbols MbRo f ts J 
Variation[NbWeights]+ = MinLen 

SplitPoint = NbSymbols * (NbWeights - (NbRoots * NbSymbols MinLen )) 
select (W[l. ■ NbWeights], SplitPoint) 
Increment Variation[SplitPoint] 
else 

CIStart = C [NbClusters] 

CISize = NbWeights - CIStart + 1 

MinLeaf = rainig[cistart..iibWiiights] 

W[i] 

SumWeights = PartialSum[ClStart - 1] 
MaxNbNodes = [SumWeights/MinLeaf ] 

m-„t I / CISize+MaxNbNodes \ I 

MmLen = |log BbSymbols ( ^^Ts )J 

NbTuples = MaxNbNodes + CISize - NbRoots * NbSymbols MinLen 
Variation[NbWeights] = MinLen 
if 2 * NbTuples < MaxNbNodes then 

TopDown(ClStart — 1, NbClusters-1, MaxNbNodes -NbTuples) 
else 

SplitPoint = ClStart+ NbSymbols*NbTuples- MaxNbNodes 
select (W, SplitPoint) 
Increment Variation[SplitPoint] 
TopDown(ClStart-l, NbClusters-1, MaxNbNodes) 
end if 
end if 



Final £)-ary algorithm: We recap our linear time algorithm for the D-ary case in the following theorem: 

Theorem 2. Given N integer weights W[1..N], and a positive integer D > 1, there is an algorithm which 
computes an optimal D-ary prefix free code for those weights in time and space linear in the input encoding 
size. 

Proof. Given N unsorted weights and a positive integer D > 1, the only difference with the algorithm from 
Theorem [1] is the call to DAryTopDown, and the technical details of the generation of the codes themselves: 

1. Sort the weights by their logarithm in base 2 (even though we are searching for a D-ary code) into 
clusters of weights all with the same logarithmic integer part in base 2, via Lemma [2] in time linear in 
the input encoding size. 

2. Compute a vector of variations in code lengths via a single call to the recursive function DAryTopDown 
with parameter values NbWeights = N, NbClusters = S, NbRoots = 1 and NbSymbols = D. 

3. Compute the code lengths assigned to the weights by summing the variations from the last position to 
the first (i.e. L[i] = J2j=i Variation[i]); 

4. Compute the codes themselves from the code lengths, using a variant of the technique described by 
Schwartz et al. [26] in time linear in the number N of weights. 

Thus, the overall complexity of our algorithm is linear in the input encoding size. 
3.2 Compressed Data Structure 

Our algorithm computes a permutation of the weights (and its runs). In this permutation, there are at most 
25 runs, the length of those runs forming a vector dominated by the vector formed by the number of weights 
with a given code length. 

Combining Barbay et aVs compressed data structures for permutations pQ with Moffat et aVs data- 
structure [IS] to map codes to code lengths and rank inside the cluster of messages with the same code 
length, yields the following corollary: 
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Corrolary 1 There is a compressed succinct data structure computed in O(N) time using N\og 2 k + 0(N) + 
o(N \og 2 n) UtE where k is the smallest possible number of distinct code lengths in an optimal prefix free 
code, which supports the operators code(m) and decode(5) in time linear in the code length plus OQogK) 
in the worst case. 

Proof. Consider the permutation tt mapping the positions of the weights in the order produced by our 
algorithm relative to their position in the original input array. A run of tt is a consecutive subsequence of 
increasing values. If a consecutive subsequence of weights in the array of weights partially ordered by our 
algorithm were in the same relative order in the original array, it will correspond to a run of equal length in 

TT. 

Since counting sort does not affect the relative order of the weights with the same integer part of their 
logarithm in base 2, and since the relative order of the weights inside each cluster is changed only by a single 
select operation, the permutation tt contains at most 28 runs, two for each cluster. The number of clusters 
of weights with the same logarithm integer part in base 2 is at most 2k by Lemma [TJ so the permutation tt 
contains at most 4k runs. 

Barbay et al. [I] showed how to encode such a permutation into N(l + log 2 4k)) + o(N(l + log 2 k))) = 
iV(3 + log 2 k)) + o(N(l + log 2 k))) bits in order to support the direct 7r() and inverse 7r _1 () mapping via 
tt in time O(logK) in the word RAM model. Their data structure implicitly indicates to which run of the 
permutation belongs an integer, and its rank in it. This translates in our application to the rank of the code 
length assigned to a message, and the rank of the message among those assigned the same code length. 

Moffat et al. [TB] showed how to map such pairs, formed by the rank of the code length and the rank 
of the messages among messages of same code length, into a code with the numerical property, using only 
O(KlogK) C o(N log k)) bits. 

3.3 Prefix Free Codes in External Memory 

Given that both the algorithm Counting Sort and the support for the select operator are easily extended 
to external memory, our techniques also yield an algorithm computing a prefix free code in 0{n) accesses to 
external memory when the input occupies n pages. 

Corrolary 2 Given N weights occupying n pages in external memory, there is an algorithm which computes 
an optimal prefix free code for them in 0(n) accesses to external memory. 

Proof. The most costly operation is the select operator: it is supported in linear number of accesses to 
external memory [2S]- All the other instructions of TopDown and DAryTopDown can be implemented via a 
scan of the weights of a cluster, which sums to an I/O complexity within 0(n). 

4 Discussion 

4.1 Practical considerations 

Computation of the Logarithm: The computation of the integer part of the logarithm in base 2 of an 
integer x is supported in a single CPU cycle on most modern architecture, under the form of counting the 
number of leading zeroes in the binary encoding of x. 

Likewise, the computation of the integer logarithm of an integer x in any base D which is a power of 
2 can be computed in exactly two CPU Cycles: one to compute the integer part of the logarithm base 2, 
and one to shift the binary encoding of the result in order to divide it by D. The code lengths are then 
all multiple of log 2 D, which gives some practical advantages: Moura et al. [22] showed in their study of 
compressed indexes for natural language data sets that such values, D = 128 = 2 7 and D = 256 = 2 s in 
their studies, yield better time for several types of queries, exact and approximated. 

For more general values of D, the computation of [log£> x\ reduces to 1 + [log^ x\ integer divisions and 
comparisons, so that computing the logarithm of k positive integers {Ni, . . . ,Nk} which sum to N reduces 
to k + ^T. [\og D Ni\ < k + N < 2N integer divisions and comparisons. 

4 Asymptotically in N. 
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Practical Support of the select operator: Rather than supporting the select operator through the 
median, a practical implementation should support it through the QuickSelect algorithm, which has a much 
better average performance. 

Space-Time trade about Partial Sums: The algorithm as we describe it is computing a vector PartialSum 
and maintaining it through all operations. Such a vector is used in order to compute the sum of the weights in 
any contiguous subsequence of W[l..iV"] in constant time, and in particular in the subsequence [L.ClStart]. 

Should the space cost of this solution make the implementation impractical, it is possible to reduce the 
space usage in two ways: 

1. Compute the sum of the weights in each cluster, and the partial sums only of the clusters themselves. 
This solutions reduces the space from N additional values to S additional values, at no time cost. 

2. Pass in parameter the sum of the NbWeights weights in the range [L.NbWeights]; compute at each 
iteration of the algorithm the sum of the weights in the range [CIStart + L.NbWeights], and deduce 
from their difference the sum of the weights in the range [L.ClStart]. This solution reduces the space 
from N values to constant, but costs more time (without affecting the asymptotic complexity). 

Sorted vs Unsorted Weights: According to Moffat et al. 21,, in practice the weights can always be 
assumed to be sorted. Their argument is that all Huffman based compression algorithms involve a mapping 
from the input component to the messages on which Huffman's code is computed, and that this mapping 
can always be chosen in such a way that the messages are sorted by their weights. But prefix free codes have 
more sophisticated applications than direct text compression: we give here one example and assume that 
there are others. 

Barbay et al. [T\ reduced the compression of a permutations with p runs (subsequences of consecutive 
positions already in order) to the encoding of a Huffman-shaped wavelet tree. The mapping in this case is 
direct: the i-th weight corresponds to the size of the i-th run. Introducing an additional mapping to sort the 
messages by weights comes at a cost of p\\og 2 p] additional bits. This is sufficiently costly to make using 
Hu- Tucker's code more advantageous, reducing the cost from /9[log 2 p] to p additional bits. 

4.2 Comparison with Related Work 

Prefix Free Codes in the worst case over N: In the 40's, many people were searching for codes 
insuring the best compression possible through prefix free codes. In 1948, independently Shannon and Fano 
defined Shanon-Fano codes [9127] . Their algorithm is building a binary tree top-down, partitioning the N 
probabilities at the root in two subsets, so that the sum of the probabilities on the left would be as close as 
possible to the sum of the probabilities on the right, and recursing on each side. Such codes are computed in 
linear time as the length of the binary sequence associated with a particular symbol can be directly deduced 
from its frequency, but they are not optimal. 

In 1951, Huffman, then a student of Fano's course at MIT, took Fano's offer to his students to skip the 
final exam if they could develop an optimal efficient prefix free code |33j . Huffman |13j proposed to build 
a tree, bottom-up, by joining the smallest probabilities and the nodes of smallest accumulated frequency 
instead of top-down as in Shannon-Fano coding. The algorithm suggested by Huffman's solution produces a 
code optimal among all prefix free codes in 0(N log N) algebraic operations for TV messages. 

In 1976, Van Leeuwen [16] showed that this code tree, and the associated code, could be constructed in 
a linear number of algebraic operations from the sorted list of probabilities, providing a simple reduction in 
linear time to ordering the probabilities of the messages. 

In 2002, Han pT] showed how to sort N integers in time O(iVloglogiV) using non algebraic operations. 
Encoding the probabilities as integers rather than fractional numbers (for instance, by counting the number 
of occurrences of the messages rather than their probability), one can combine this result with Van Leeuwen's 
reduction |16j to yield an algorithm computing an optimal prefix free code in the same asymptotic time. 
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Our work produces another code than the one originally defined by Huffman [13] : as such, it does not 
improve on the complexity of Huffman's algorithm in the strictest sense. Nevertheless, the essence of Huff- 
man's result was not so much the code that he described than the existence of optimal prefix free codes and 
a method to compute one of them in reasonable time. Our result improves on Huffman's in that it produces 
another optimal prefix free code in less time, with additional properties of interest. 

Given that our algorithm outputs an optimal prefix free code in linear time, it removes whatever advantage 
kept by the Fano-Shannon codes, which can be computed in linear time but are suboptimal. 

To the extent of our knowledge, 0(N log log N) is the lowest complexity [35] known previous to ours, in the 
worst case over all possible input of N. Our result improves this complexity complexity from O(iVloglogiV) 
to 0{N), 

Canonical Codes: In 1964, Schwartz et al. |26j . introduced the concept of canonical code in which "the 
numerical values of the codes are monotone increasing and each code has the smallest possible numerical 
value consistent with the requirement that the code is not the prefix of any other code" . They described how 
to compute such codes by computing first the Huffman's code, then deducing from it an optimal vector 
of code lengths, and then computing from the later an optimal prefix free canonical code, "iteratively by 
assigning codes to the words in order of decreasing rank" . 

In 1973, Connell [5] defined the Huffman- Shannon- Fano code combining the optimality of Huffman's code 
with the ease of encoding and decoding of Shannon-Fano's codes. This code has the same redundancy than 
Huffman's code and holds the numerical sequence attribute of the Shannon-Fano, which requires that, when 
the messages are ordered by their weights and the binary codes are in lexicographical order, the codes of 
same code length must correspond to consecutive weights. They claim that this new code reduces "translation 
table size, encoding time, and decoding time. It should also simplify any digital circuit designed specifically 
for the translation process" . 

In 1997, Moffat et aL[SU] further reduced the time of the decoding process to be linear in the number of 
output message rather than the number of received symbols. This process was further improved in 2005 by 
Tjalkens [30]. 

A first comparison of our results with previous results on Canonical codes 26 6 20 30 is that the algo- 
rithms producing those codes either assume that the weights of the messages are already sorted, or compute 
a Huffman code in 0{N log N) time, when ours is running in 0{N) time. A second comparison is that our 
solution is more direct: our algorithm directly builds a canonical code from the unsorted weights removing 
many intermediary steps, and minimize the perturbation of the order of the weights. 

Prefix Free Case in Sorted Particular Cases: In 1998, Moffat et al. \21[ showed how van Leeuwen's 
algorithm 16 (which works from a sorted sequence of weights) can be made faster when the number r of 
distinct weights in the input is small, which yields an algorithm performing 0(r(l + log(N/r)) algebraic 
operations in the worst case over fixed values of N and r. 

Note that this adaptive result is on sorted input. In particular, it should not be confused with other 
potential adaptive results based on repeated values, such as an (adaptive) algorithm running in time within 
0(N(1 + H(f\, . . . , f r ))) given N weights taken from r distinct values, where the i-th value is present fi 
times, obtained by sorting [23] the weights in time 0(N(1 + %{f\, ■ ■ ■ , fr))) and computing [16] from the 
sorted weights an optimal prefix free code in time 0{N). 

In 2001, Milidiu et al. [17] showed how clustering similar weights permits to reduce the space usage 
of linear time algorithms computing binary prefix free codes from sorted weights. They state a result |17[ 
Proposition 3] similar to Lemma Q] (albeit they use it in the sorted case, and we use ours in the unsorted 
case): given a sorted list W[l] < . . . < W[iV] of weights, if there is an integer i G [1..N/2] such that the 2i-th 
smallest weight VF[2i] is smaller than the sum W[l] + W[2] of the two smallest weights, then there exists an 
optimal binary code tree where the nodes corresponding to the 2i smallest weights are paired two by two in 
order, i.e. W[2j - 1] and W[2j] are sibling, Vj e [0..i - 1]. 

In 2006, Belal et al. [3J, described a variant of Milidiu et a/.'s algorithm [T7], announcing that it performed 
©(log 2 ^ 1 N) when the weights are sorted, where k is the number of distinct code lengths in some optimal 
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prefix free code. Note that k is not well defined: there exist sets of weights W[l..iV] for which exist optimal 
prefix free codes with a number of distinct code lengths varying from simple to double (i.e. from a minimal 
value of k to a maximal value of 2k). Van Leeuwen's algorithm ^16 insures the minimization of the maximal 
code length by giving preferences to leaves over nodes of same weight, but does not yield a code neither with 
the minimal nor the maximal number of code lengths. 

The techniques used by Moffat et al. [3T] and Milidiii et al. [17] greatly inspired ours, as well as the 
examples given by Belal et al. [3J, even though their work (and Belal et aL's examples) is set in the sorted 
case whereas our work is set in the unsorted case. 

Prefix Free Case in Unsorted Particular Cases: In 2006, Belal et al. [3J, described a variant of 
Milidiii et al.'s algorithm [17], announcing that it performed O(kN) algebraic operations when the weights 
are not sorted, where k is, as in the previous section, the number of distinct code lengths in some optimal 
prefix free code. 

They describe through examples an algorithm claimed to run in time 0(4 k N) when the weights are 
unsorted, and propose to improve the complexity to O(kN) by partitioning the weights into smaller groups, 
each corresponding to disjoint intervals of weights valued- The claimed complexity is asymptotically better 
than the one suggested by Huffman when k £ o(log N), and they conjectured the existence of an algorithm 
running in time 0(N log k). 

Like our algorithm, the algorithms evocated by Belal et al. [3J for the unsorted case are based on several 
computations of the median of the weights within a given interval, in particular in order to select the weights 
smaller than some well chosen value. The essential difference is that given that their algorithm constructs 
the code "bottom-up" by selecting first the message of smaller weights, following the model of Huffman's 
original solution, it ends up potentially moving several nodes from the level where it was originally assigned. 
Our solution avoids this cost by computing "top-down" the code length assigned to each message once for 
all. 

4.3 Impact of our Results 

We described techniques to compute an optimal prefix free code in time linear in the input encoding size, 
improving on the previous solutions which were taking time more than linear. 

Relevance of Optimal Prefix Free codes: Albeit 60 year old, Huffman's result is still relevant nowadays. 
Optimal Prefix Free codes are used not only for compressed encodings, but also for compressed data structures 
and for organizing the input so that to speed-up sorting algorithms pQ. 

In 1991, Gary Stix |29j stated that "Large networks of IBM computers use it. So do high- definition tele- 
vision, modems and a popular electronic device that takes the brain work out of programming a videocassette 
recorder. All these digital wonders rely on the results of a 40-year-old term paper by a modest 
Massachusetts Institute of Technology graduate student- a data compression scheme known as Huffman 
encoding (...) Products that use Huffman code might fill a consumer electronics store. A recent 
entry on the shop shelf is VCR Plus+, a device that automatically programs a VCR and is making its in- 
ventors wealthy. (...) Instead of confronting the frustrating process of programming a VCR, the user simply 
types into the small handheld device a numerical code that is printed in the television listings. When it is 
time to record, the gadget beams its decoded instructions to the VCR and cable box with an infrared beam like 
those on standard remote- control devices. This turns on the VCR, sets it (and the cable box) to the proper 
channel and records for the designated time." . 

In 1995, Moffat et al. [18j . stated that: 11 The algorithm introduced by Huffman for devising minimum- 
redundancy prefix free codes is well known and continues to enjoy widespread use in data compression 
programs. Huffman's method is also a good illustration of the greedy paradigm of algorithm design and, at the 

5 Note that several of those results were downgraded in the December 2010 update of their initial 2005 publication 
through Arxiv [2]. 
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implementation level, provides a useful motivation for the priority queue abstract data type. For these reasons 
Huffman's algorithm enjoys a prominence enjoyed by only a relatively small number of fundamental 
methods" . 

In 1997, Moffat et al. 20 stated that those were "one of the enduring techniques of data com- 
pression. It was used in the venerable PACK compression program, authored by Szymanski in 1978, and 
remains no less popular today" . 

In 2010 Donald E. Knuth was quoted [24], as saying that: "Huffman code is one of the fundamental 
ideas that people in computer science and data communications are using all the time" . 

In 2010, the answer to the question "What are the real-world applications of Huffman coding?" on the 
website Stacks Exchange states that "Huffman is widely used in all the mainstream compression formats 
that you might encounter - from GZIP, PKZIP (winzip etc) and BZIP2, to image formats such as JPEG 
and PNG." [31]. 

The Wikipedia website on Huffman coding states that "Huffman coding today is often used as a "back- 
end" to some other compression method. DEFLATE (PKZIP 's algorithm) and multimedia codecs such as 
JPEG and MPS have a front-end model and quantization followed by Huffman coding." |33j . 

Ironically, the pseudo-optimality of this algorithm seems to have become part of the folklore of the 
area, as illustrated by a quote from Parker et al. [5] in 1999: " While there may be little hope of improving 
on the 0(N log N) complexity of the Huffman algorithm itself, there is still room for improvement in our 
understanding of the algorithm." . 

Practical Applications of our results: The impact of our faster algorithm on the execution time of 
optimal prefix free codes based techniques should definitely be evaluated further. Yet we do not expect it to 
be meaningful: compressing a sequence S of \S\ messages from an input alphabet of size N requires not only 
to compute the code (in time 0{N) using our solution), but also to compute the weights of the messages (in 
time S\), and to encode the sequence S itself using the computed code (in time OdS"!)). In most application, 
improving the code computation time will not improve on the compression time. 

Rather, we expect that the most important impact of our results will come from the code itself which is 
computed, and in particular from the permutation defined by the reordering of the weights. This permutation 
is intuitively the permutation closest to the identity that still yields a canonical code, and as such is highly 
compressible, which yields a compressed representation of the code. We expect that in applications (e.g. 
networking cards) where an optimal prefix free code is built in an electronic circuit, the code produced 
by our algorithm will yield smaller and faster solutions than more traditional codes such as the one from 
Huffman|13) or Schwartz [26] . 

The next logical steps are 

1. to check, among the compression algorithms computing an optimal prefix free code on each new instance 
(e.g. JPEG, MP3, MPEG), which see their time performance improved by a faster prefix free code 
algorithm; and 

2. to check, among the communication solutions using an optimal prefix free code computed offline, which 
can now afford to compute a new optimal prefix free code more frequently and see their compression 
performance improved by a faster prefix free code algorithm. 

Related Problems The techniques described in this work should find applications to other problems which 
are similar to the computation of optimal prefix free codes, such as Bounded Length Optimal Prefix Free Codes 
and Alphabetic Binary Search Trees. 

Given N positive integer weights W[l..iV], a number D of output symbols,, and a positive integer B > 
[log 2 N~\ , a Bounded Length Optimal Prefix Free Codes [19] is a prefix free code optimal among all the 
prefix free codes with maximal code lenght B. The best complexity known for this problem seems to be 
0(BN log N) via a dynamic programming argument |19j . Yet as for optimal prefix free codes, the bounded 
version can be computed in linear time when all the weights are within a factor of less than two of each 
other: it's intriguing to know if our TopDown algorithm can be adapted to this case. 
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Given N positive integer weights W[1..JV], an optimal Alphabetic Search Trees [TS] is a set of N code 
strings of variable lengths L[1..N] and on alphabet [1..D] such that not only no string is prefix of another 
(i.e. ^2j—i D~ L ^ — 1) and the average length of a code is minimized (i.e. X^i^H^M is minimal), but 
also such that the lexicographical order of the codes is the same as the original order of the weights. The 
best complexity known is 0(N log N) via the Hu- Tucker [T5] algorithm, which is very similar to the one used 
to computed Huffman's codes. As for optimal prefix free codes, an optimal alphabetical search tree can be 
computed in linear time when all the weights are within a factor of less than two of each other: it seems likely 
that the integer part logarithms of the weights will similarly be usefull to compute an optimal alphabetical 
search tree in the general case. 
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